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Collaboration  and  Knowledge  Interoperability  Program 


Individual  Knowledge 
Building 


Developing  Knowledge 
Interoperability 


Work  by  teams  whose  members  are 
separated  by  space  and  time. 


Program  Objective 

Understand  the  cognitive  processes 
underlying  team  decision  making  in 
order  to  aid  and  improve  team 
performance  in  quick-reaction, 

NDM-type  problem  solving. 


Team  Consensus 
Development 


Attaining  Shared 
Understanding 


1 


3.  #  4 


WHAT’S  NEW  IN  TEAMS  ? 


Advanced  and  ubiquitous  socio-technical  systems 
Globalization  and  distribution  of  team  members 
Multidisciplinary  members 
Ad-hoc  problems  with  increasing  complexity 


Accelerated  speed  of  response 


Transformational  Team  Characteristics 


•  Unstructured,  agile  teams 

•  Distributed  and  asynchronous  relationships 

•  Heterogeneous  members  (multidisciplinary,  multicultural) 

•  Ad-hoc,  NDM  type  decision  making  and  problem  solving 

•  Short  duration,  high  stress  problems 

•  Uncertainty  in  source  information 

•  Dynamic  information 

•  Rotating  team  members 

•  Technology  supported/collaborating  agents 


MACROCOGNITION  IN  TEAMS  -  A  definition 


Macrococinition  is  defined  as  the  internalized  and  externalized  high-level  mental 
processes  employed  by  teams  to  create  new  knowledge  during  complex,  one-of-a-kind, 
collaborative  problem  solving. 

High-level  is  defined  as  the  process  of  combining,  visualizing,  and  aggregating  information  to  resolve 
ambiguity  in  support  of  the  discovery  of  new  knowledge  and  relationships 

Internalized  processes  (processes  occurring  inside  the  head)  are  those  higher-level  mental  [cognitive] 
processes  that  occur  at  the  individual  or  the  team  level,  and  are  not  expressed  externally  (e.g.  writing, 
speaking,  gesture),  and  can  only  be  measured  indirectly  via  qualitative  metrics  (e.g.,  questionnaires, 
cognitive  mapping,  think  aloud  protocols,  multi-dimensional  scaling,  etc.),  or  surrogate  quantitative 
metrics  (e.g.,  pupil  size,  galvanic  skin  response,  fMRI).  These  processes  can  become  either  fully  or 
partially  externalized  when  they  are  expressed  in  a  form  that  relates  to  other  individual’s  reference/ 
interpretation  systems  (e.g.  language,  icons,  gestures,  boundary  objects) 

Externalized  processes  (processes  occurring  outside  the  head)  are  those  higher-level  mental 
[cognitive]  processes  that  occur  at  the  individual  or  the  team  level,  and  which  are  associated 
only  with  actions  that  are  observable  and  measurable  in  a  consistent,  reliable,  repeatable 
manner  or  through  the  conventions  of  the  subject  domain  have  standardized  meanings. 

Externalized  processes  can  be  measured  by  various  techniques  each  with  their  associated 
metrics,  which  include  discourse  analysis,  process  tracing,  automated  latent  semantic 
analysis,  automated  communication  flow  analysis,  and  dynamic  modeling  of  communication 
data 


WHY  MACROCOGNITON  ? 


Within  the  field  of  cognitive  engineering,  theorists  have  proposed 
the  term  “macrocognition”  to  describe  how  cognition  emerges  in  natural 
environments.  This  line  of  thinking  is  essentially  based  upon  the  work  of 
Cacciabue  and  Hollnagel  (1995)  who  specifically  noted:  “Macrocognition 
refers  to  the  study  of  the  role  of  cognition  in  realistic  tasks,  that  is,  in 
interacting  with  the  environment”  (p.  57).  Klein  and  colleagues  have 
continued  to  argue  that  contextually  bound  cognitive  processes  (e.g., 
sense  making,  uncertainty  management)  must  be  studied  in  natural 
settings  (Klein  et  al.,  2003).  These  are  environments  in  which  complex 
and  emergent  cognitive  processes  arise  (i.e.,  macrocognitive  processes), 
as  opposed  to  “micro-cognitive”  processes  described  as  cognition  used 
in  laboratory  studies.  We  extend  this  theorizing  and  adopt  the  more 
recent  thinking  on  macrocognition  in  teams.  Here  the  term  is  used  to 
capture  cognition  in  collaborative  contexts.  In  their  theoretical  and 
empirical  analysis  of  collaborative  problem  solving,  Warner,  Letsky,  and 
Cowen  (2005)  argue  that  macrocognition  in  teams  encompasses  both 
internalized  and  externalized  processes,  which  occur  during  team 
interaction 


MACROCOGNITION  -  Complementary  Perspectives 


MACROCOGNITION  vs 

(in  situ) 

MACROCOGNITION 
(in  teams) 

•  Natural  settings 

•  lab-based  /  experimental 

•  Task  specific 

•  Extensible 

•  CTA  -  based 

•  Metrics-based 

•  Practitioners 

•  Researchers 

•  Holistic  approach 

•  Reductionist  approach 

•  Product  focused  /  goal  directed 

•  Phenomena  focused 

NDM  /  RPD 


Conceptual  model 


MACROCOGNITIVE 


Individual  Knowledge  Building:  Individual  team 
members  ask  for  clarification  of  data  or  information,  or 
respond  to  clarification  requested  by  other  team 
members. 

Team  Knowledge  Building:  All  team  members 
participate  in  clarifying  information  (e.g.,  answering  a 
question)  to  build  team  knowledge. 

Developing  Shared  Problem  Conceptualization:  Team 
members  sharing  their  understanding  of  problem  goals, 
characteristics  of  the  environment  and  rules  of  operating 
for  the  generation  of  quality  problem  solutions. 

Team  Consensus  Development:  Team  negotiation  of  a 
solution  option  and  collective  agreement  by  team 
members  on  a  particular  option  (i.e.  each  team  member 
does  not  have  to  agree  on  the  solution  option,  but  as  a 
team  they  need  to  agree  on  the  final  option  selected). 

Outcome  Appraisal:  Team  evaluation  (all  team 
members)  of  selected  solution  option  against  problem 
solving  goal.  Team  revises  solution  option  if  option  does 
not  meet  goal. 


Individual  Knowledge  Building 

■  Iterative  Information  Collection 

■  Individual  Task  Knowledge  Development 

■  Individual  Mental  Model  Development 
Team  Knowledge  Building 

■  Team  Pattern  Recognition  and  Trend  Analysis 

■  Team  Mental  Model  Development 

■  Recognition  of  Expertise 

■  Sharing  Unique  Knowledge 
■Uncertainty  Resolution 

■  Knowledge  Interoperability 


Developing  Shared  Problem  Conceptualization 

*  Visualization  and  Representation  of  Meaning 

*  Building  Common  Ground 

*  Knowledge  Sharing 

*  Knowledge  Transfer 

■  Team  Shared  Understanding 


Team  Consensus  Development 

*  Critical  Thinking 

*  Mental  Simulation 

*  Intuitive  Decision  Making 

*  Iterative  Information  Collection 

•  Solution  Option  Generation 

•  Storyboading 

•  Team  Pattern  Recognition  &  Trend  Analysis 

*  Team  Negotiation  of  Solution  Alternatives 
Outcome  Appraisal 

•  Feedback  Interpretation 

•  Replanning 

•Team  Pattern  Recognition  &  Trend  Analysis 


METRICS 


Cognitive  maps 
Discourse  analysis 
Think  aloud 

Latent  Semantic  Analysis 
Communication  flow  analysis 
Dynamic  modeling  of  communication 
Card  sorting 
Pre/post  questionnaires 


RESEARCH  CHALLENGES  /  APPROACH 


•  REFINE  CONCEPTUAL  MODEL 

•  IDENTIFY  AND  OPERATIONALIZE  PROCESSES 

•  APPLY  METRICS  TO  EMPIRICALLY  VALIDATE  CONSTRUCTS 

•  PREDICT  IMPACT  ON  TEAM  PERFORMANCE 


MAKE  FINDINGS  EXTENSIBLE 


STEPS  FORWARD 


MULTI-TRAIT  MULTI-METHOD  APPROACH 

TECHNOLOGY  DEMOS  OF  PROCESS  AIDS 

TESTBEDS  /  SCENARIOS 

INTERIM  TOOLS 


GUIDELINES,  MODELS 


BACK  UP  SLIDES 


CKI  Program  Structure 


(FY06/07) 


070106 


Representation  and  Transfer  of  Meaning 

•  Electronic  Card  Wall  (EWALL)  MIT 

•  Collaboration  and  Meaning  Analysis  Process  (C-MAP)  UTenn 

•  Decision  Making  Constructs  in  a  Distributed  Environment  (DCODE)  SPAWAR 

•  Communication  in  Team  Cognition  (COMTC)  ASU 

•  Augmented  Reality  Visualizations  for  Shared  Understanding  (AUGVID)  UCSD 

•  Sys  for  Understanding  and  Measuring  Macrocognition  in  Teams  (SUMMIT)  MURI  UCF 

Attaining  Shared  Understanding 

•  Mental  Model  Convergence  (MMC)  UMASS 

•  Common  Ground  in  Geo-Collaboration  (CGGC)  Penn  State 

•  Automated  Comm  Analysis  for  Interactive  SA  Assessment  (ASA)  04-119  SBIR  SA  Tech 

•  Instrument  for  Measuring  and  Advancing  Group  SA  (IMAGES)  04-T026  STTR  Aptima 

•  Human-Centric  Architecture  for  Net-Centric  Operations  (DCCS)  04  -T026  STTR  PSE 

•  Processes  in  Complex  Team  Problem  Solving  (TPS)  UCF 

Team  Consensus  Building 

•  Collaborative  Critical  Thinking  (CCT)  (CENTER)  00-086  SBIR  Aptima 

•  Macro  Cognition  in  Team  Collaboration  (MCTC)  NAVAIR 

•  Shared  Information  Virtual  Surfaces  (SIVS)  Col.  State 

•  Joint  Intelligence  Graphical  Situation  Awareness  Web  (JIGSAW)  04-116  SBIR  PSE 

•  Enhancing  Tactical  Decision-Making  in  Navy  Seal  Ops  (SLATE)  05-069  SBIR  PSE 

•  Integrated  Dec  Space  Model  for  Cultural  Diff  (IDSM)  (IDecS)  03-112  SBIR  PSE 

•  NPS  Testbed  for  Team  Collaboration  (NPSTB)  NPS 


COLLABORATION  AND  KNOWLEDGE  MANAGEMENT  (CKM)  PROGRAM 

MODEL  OF  TEAM  COLLABORATION 
FOCUS  ON  MACRO-COGNITON 

Problem  Area 
Characteristics 


■  time  pressure 

•  information/knowledge 
uncertainty 

•  dynamic  information 

•  large  amount  of  knowledge 
(cognitive  overload) 

•  human-agent  interface 


asynchronous 

distributed 

culturally  diverse 

heterogeneous  knowledge 

unique  roles 

command  structure 
(hierarchical  vs.  flat) 

>  rotating  team  members 


•  team  decision  making,  COA 
selection 

•  develop  shared  understandir 

•  intelligence  analysis 
(team  data  processing) 


Collaboration  Stages  &  Cognitive  Processes 


Meta-Cognitive: 

•  individual  conversion  of 
data  to  knowledge 


MacroCognitive: 


•  individual  mental  model 
constmction 

•  knowledge  interoperability 
development 

•  individual  task  knowledge 
development 

•  team  knowledge  development 

•  individual  knowledge  object 
development 

•individual  visualization  and 
representation  of  meaning 


•  team  integration  of  individual  •  team  agreement  on  a  common 

knowledge  for  common  understanding  solution 


•  knowledge  interoperability 
development 

•  iterative  information  collection 

and  analysis 

•  team  shared  understanding 
development 

•  develop,  rationalize,  &  visualize 
solution  alternatives 

•  convergence  of  individual  mental 
models  to  team  mental  model 

•  individual  task  knowledge 
development 

•  team  knowledge  development 


I 


•  team  negotiation  of  solution 
alternatives 

•  team  pattern  recognition  I 

•  team  shared  understanding  \ 
development 

•  convergence  of  individual  ; 

mental  models 

•  critical  thinking 

•  sharing  hidden  knowledge 

•  individual  task  knowledge 
development 

•  team  task  knowledge 
development 


•  solution  adjustment  to 
fit  goals  and  exit  criteria 


•  compare  problem  solution 
against  goals 

•  team  shared  understanding 
development 

•  convergence  of  individual 
mental  models  of  solution 


i 

j  •  analyze,  revise  output 

I 

i 


i 


Mechanisms  for  achieving  Meta  and  Macro  Cognitive  Processes  (applies  to  all  stages) _ 

•  Verbal  eonnnuni cationspmsmti no  and  discussing  individual  information,  discussing  team  generated  information 
questioning,  agreeing  /  disagreeing,  negotiating  perspectives,  discussing  possible  solutions,  providing  rationale 

•  Non-Verbal  communicationsfacial  expressions,  voice  clues  (vocal  paralanguage),  band  gestures,  body  movements  (kinesics 
touch  (haptics),  personal  space,  drawing,  text  messages,  augmented  video,  affordances  (cognition  in  objects). 


Defense  Transformation  /  NCO  /  FORCEnet 


A  Need  for  Agile  Teams 


Fifteen  percent  increase  in  U.S.  Special  Operations  Forces  (SOF)  including  secret  Delta  Force 
operatives  skilled  in  counterterrorism. 

A  one-third  increase  in  Army  Special  Forces  battalions,  whose  troops  are  trained  in  languages 
and  to  work  with  indigenous  fighters 

"SOF  will  increase  their  capacity  to  perform  indirect  and  clandestine  operations  in  politically 
sensitive  environments  and  denied  area 

"SOF  will  have  the  capacity  to  operate  in  dozens  of  countries  simultaneously"  and  will  build 
relationships  with  "foreign  military  and  security  forces," 

The  creation  of  small  teams  of  operatives  assigned  to  "detect,  locate,  and  render  safe"  nuclear, 
chemical  and  biological  weapons  --  as  well  as  to  prevent  their  transfer  from  terrorist  groups. 


Quadrennial  Defense  Review 


Significance  for  Team  Decision  Making 


Asymmetric  warfare,  including  increasing  information  uncertainty, 
increasing  multidisciplinary  and  multicultural  participants  and 
complex  political  considerations  will  dictate  more  human  analysis 
and  intervention. 

As  decision  making  becomes  more  time-compressed,  individuals 
and  teams  will  have  to  operate  with  less  (technology  and  human) 
advisory  support. 


Quick  response,  agile,  mixed  discipline  teams  will  become  an 

important  force  multiplier 


Research  Challenge: 

Understand  and  Improve  Transitional  Team  Performance 


Defense  Transformation  /  NCO  /  FORCEnet 


....  information  operation  and  networking  and  connecting  things  in  ways  that  they 

function  totally  differently  than  they  had  previously  . if  that's  possible,  the 

single-most  transforming  thing  in  our  force  will  not  be  a  weapon  system,  but  a  set 
of  interconnections  and  a  substantially  enhanced  capability  because  of  that 
awareness. “  -  Secretary  of  Defense  Donald  H.  Rumsfeld 


"If  you  are  not  interoperable ,  you  are  not  on  the 
net,  you  are  not  contributing,  you  are  not 
benefiting,  and  you  are  not  part  of  the  information 
age  -  Vice  Adm  Art  Cebrowski  [DoD  Director, 

Force  Transformation 


Network 
Centric 
Operations 
The  Way  Ahead 


Collaboration  technologies  assist  the  operator  in  making  sense  out  of  the  data  he 
or  she  pulls.  To  successfully  operate  within  our  increasingly  complex  and 
interrelated  world,  unprecedented  combinations  of  subject  matter  experts  are  often 
needed  to  make  sense  out  of  special  situations.  These  experts  are  not  likely  to  be 
found  in  any  one  single  unit  or  organization.  Therefore,  the  ability  to  pull  expertise 
from  both  within  a  unit  as  well  as  from  across  the  Department  is  an  important 
value-added  feature  of  net-centricity. 


